Skip to content

Add ability to enforce concurrent query limits#3257

Open
lbschanno wants to merge 51 commits intointegrationfrom
task/queryLimit
Open

Add ability to enforce concurrent query limits#3257
lbschanno wants to merge 51 commits intointegrationfrom
task/queryLimit

Conversation

@lbschanno
Copy link
Collaborator

@lbschanno lbschanno commented Oct 31, 2025

Configuration

Add the ability to enforce concurrent query limits across a group of webservers. Zookeeper is used to track active queries and the following data:

  • The query ID
  • The user who submitted the query
  • The system the query was submitted on
  • The query logic the query originated from

Query limit enforcement is done through the QueryLimiter class. Given a user, system, and query logic, it can determine if any of the following limits have been exceeded:

  • The max allowed concurrent queries for the user.
  • The max allowed concurrent queries of the query logic for the user.
  • The max allowed concurrent queries for the system.
  • The max allowed concurrent queries of the query logic for the system.

Limits may be defined and customized on a per-user and per-system basis. They may also be defined for groups of query logics. The classes UserLimitProvider, SystemLimitProvider, and QueryLogicGroupLimitProvider are respectively responsible for identifying the best limits to enforce for a user, system, and query logic. They will be initialized in the QueryLimiter after providing a QueryLimitConfiguration instance. The following can be configured:

On a system-wide basis:

  • The default concurrent user query limit. This applies to the total number of queries a user may run across all systems. May be overridden per user.
  • The default concurrent system query limit. Primarily to avoid a system getting overloaded. May be overridden per system.

On a per-system basis:

  • The system name/ids the configuration targets. Regex matching is supported. Pattern uniqueness is enforced.
  • The concurrent system query limit. Overrides the system-wide value.
  • Whether queries submitted to the system count towards a user's concurrent query total. This is always true unless specified otherwise.
  • The concurrent system query limit for different query logic groups. Regex matching against group names is supported. Pattern uniqueness is enforced.

On a per-user basis:

  • The user DN.
  • The user's concurrent query limit. Overrides the system-wide configuration.
  • The user's concurrent query limit for different query logic groups. Regex matching against group names is supported.

On a per-query-logic-group basis:

  • The group name.
  • The query logics included in the group. Regex matching is supported. Pattern uniqueness is enforced.
  • The default concurrent user query limit. This applies to the total concurrent queries a user may run that originate from a query logic in the group across all systems.

Given the possibilities for exact matches, partial regex matches, and wildcard regex matches, the determination of the best limit to use for any particular system, query logic, or query logic group is done by sorting matches into the following 'matching buckets' (in best-match priority):

  1. Exact match: We attempt to find an exact match first and use the associated limit.
  2. Partial regex (non-wildcard-only): If we cannot find an exact match, then we attempt to find all partial matches, and see if any of their limits are met.
  3. Wildcard-only regex: In the case of no exact or partial matches, we use the wildcard match with the lowest limit.

Implementation

Checking limits and marking as active/inactive is done through the QueryLimiter class. The three main methods to know are:

  • QueryLimiter.checkForLimits()
  • QueryLimiter.countQueryTowardsLimits()
  • QueryLimiter.stopCountingQueryTowardsLimits()

When a query is marked as active via QueryLimiter.countQueryTowardsLimits(), it will delegate to the ActiveQueryTracker, which will in turn create nodes in Zookeeper under the ActiveQueries namespace. When ActiveQueryTracker.trackQuery() is called, the following nodes are created:

# Container nodes
/users/<userDn>/<queryLogic> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>
/distinctQueryLogics/<queryLogic>  [Only created if it does not exist.]

# Ephemeral nodes. These will auto-delete themselves if their associated Zookeeper connection ever goes down.
/users/<userDn>/<queryLogic>/<queryId> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>/<queryId>

ActiveQueryTracker.trackQuery() will return a QueryHeartbeat that contain a list of PersistentNode (provided by the Apache Curator library) wrappers around the ephemeral nodes listed above. The QueryHeartbeat will maintain the connection to Zookeeper and attempt to keep the ephemeral nodes present in Zookeeper until QueryHeartbeat.stop() is called. If QueryHeartbeat.stop() is called, or the webserver crashes, the ephemeral nodes will automatically be deleted by Zookeeper.

The following error codes have been added:

412-20  - Concurrent query limit exceeded
500-164 - Error checking concurrent query limits

Closes #3100

Add the ability to enforce concurrent query limits across a group of
webservers. Zookeeper is used to track active queries and the following
data:
- The query ID
- The user who submitted the query
- The system the query was submitted on
- The query logic the query originated from

When the `ActiveQueryTracker` is instructed to track a query, the
following nodes will be created in Zookeeper under the 'ActiveQueries'
namespace:

```
/users/<userDn>/<queryId>
/systems/<systemName>/<queryId>
/queryLogics/<queryLogic>/<queryId>
/queries/<queryId>
/queries/<queryId>/user           [data = byte[] value of userDn]
/queries/<queryId>/system         [data = byte[] value of systemName]
/queries/<queryId>/queryLogic     [data = byte[] value of queryLogic]
/queries/<queryId>/heartbeats
```

This is done through the use of the `ActiveQueryTracker` class. In
addition to managing the nodes that record information about the query,
the `ActiveQueryTracker` class is also responsible for providing
instances of the `QueryHeartbeat` class.

A `QueryHeartbeat` is a wrapper around an ephemeral PersistentNode,
provided by the Apache Curator library. As long as this node is present
in Zookeeper for a particular query, the query will be considered to be
active. Should the webservers fail over and the Zookeeper connection
drop, these heartbeat nodes will automatically be deleted by Zookeeper.

The `ActiveQueryTracker` is also responsible for providing instances of
the `ActiveQuerySnapshot` class, which represent a snapshot of total
active queries at a point in time that are associated with a particular
user, system, or query logic.

Query limit enforcement is done through the `QueryLimiter` class. Given
a user, system, and query logic, it can determine if any of the
following limits have been exceeded:

- The max allowed concurrent queries for the user.
- The max allowed concurrent queries of the query logic for the user.
- The max allowed concurrent queries for the system.
- The max allowed concurrent queries of the query logic for the system.

Limits may be defined and customized on a per-user and per-system basis.
They may also be defined for groups of query logics. The classes
`UserLimitProvider`, `SystemLimitProvider`, and
`QueryLogicGroupLimitProvider` are respectively responsible for
identifying the best limits to enforce for a user, system, and query
logic. They will be initialized in the `QueryLimiter` after providing a
`QueryLimitConfiguration` instance. The following can be configured:

On a system-wide basis:
- The default concurrent user query limit. This applies to the total
  number of queries a user may run across all systems. May be overridden
  per user.
- The default concurrent system query limit. Primarily to avoid a system
  getting overloaded. May be overridden per system.
- The default of whether queries submitted to a system are counted
  towards the user's concurrent query total. This is always true.

On a per-system basis:
- The system name/ids the configuration targets. Regex matching is
  supported.
- The concurrent system query limit. Overrides the system-wide value.
- Whether queries submitted to the system count towards a user's
  concurrent query total. Overrides the system-wide value.
- The concurrent system query limit for different query logic groups.
  Regex matching against group names is supported.

on a per-user basis:
- The user DN.
- The user's concurrent query limit. Overrides the system-wide
  configuration.
- The user's concurrent query limit for different query logic groups.
  Regex matching against group names is supported.

On a per-query-logic-group basis:
- The group name.
- The query logics included in the group. Regex matching is supported.
- The default concurrent user query limit. This applies to the total
  concurrent queries a user may run that originate from a query logic in
  the group across all systems.

Given the possibilities for exact matches, partial regex matches, and
wildcard regex matches, the determination of the best limit to use for
any particular system or query logic is done by sorting matches into the
following 'matching buckets' (in best-match priority):

1. Exact match
2. Partial regex (non-wildcard-only)
3. Wildcard-only regex

and then selecting the lowest limit from the best bucket where we first
found a match.

Currently the `QueryLimiter` is used in `QueryExecutorBean`, along with a
`QueryHeartbeatCache` instance to cache heartbeats and keep them alive
when a running query is cached for retrieval later. For the purposes of
this feature, a query is considered to start when an Accumulo connection
is retrieved from the connection factory, and is considered to end when
the connection is returned to the factory.

The following error codes have been added:
412-20  - Concurrent query limit exceeded
500-164 - Error checking concurrent query limits

Closes #3100
@lbschanno lbschanno marked this pull request as ready for review December 12, 2025 22:05
lbschanno and others added 3 commits January 6, 2026 11:29
Co-authored-by: foster33 <84727868+foster33@users.noreply.github.com>
Co-authored-by: foster33 <84727868+foster33@users.noreply.github.com>
@lbschanno lbschanno requested a review from ivakegg January 22, 2026 15:42
Copy link
Collaborator

@ivakegg ivakegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like what I am seeing in here. I am thinking about potential error scenarios where things could get out of sync. Currently we have the following maps related to queries:

  1. QueryCache: A map of query id to RunningQuery instances. Used by QueryExpirationBean to check for expired queries.
  2. HeartbeatCache: A map of query id to heartbeat.

It might be worth adding a loop in the QueryExpirationBean that goes through the ids in the HeartbeatCache to verify that those queries are still running per the QueryCache. This would help to prevent anything getting out of sync if we have some unknown error cases that are not being accounted for. Other than that I am good to start integration testing this.

@lbschanno
Copy link
Collaborator Author

@ivakegg understood, I'll add in a loop to the QueryExpirationBean for synchronization with the heartbeat cache.

@lbschanno
Copy link
Collaborator Author

Added synchronization safeguard.

@lbschanno lbschanno requested a review from ivakegg January 27, 2026 00:03
@ivakegg
Copy link
Collaborator

ivakegg commented Mar 3, 2026

So I got this up and running and watching the zookeeper entries I realized that there has been one requirements which was not translated correctly and I apologize for not realizing this earlier. The "system" limit is supposed to be base on the "systemFrom" query parameter and not the hostname on which the query is running. I will take a shot at modifying the code accordingly.

@ivakegg ivakegg added the linked label Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Limit the total number of queries a user can run concurrently in the system

5 participants